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INTRODUCTION 


Evaluating  pilot  performance  is  a  difficult  and  complicated  task  due  to  the 
dynamic  nature  of  the  aviation  environment.  Throughout  history,  various  efforts  have 
explored  the  best  techniques  and  performance  measurements  to  effectively  measure  flight 
performance.  Subjective  and  objective  evaluations  have  traditionally  been  used  to 
evaluate  performance  (Rehmann,  1982).  Subjective  evaluations  often  receive  criticism 
because  of  the  potential  for  observer  biases  and  variability  amongst  observers.  Also,  it  is 
difficult  to  quantify  subjective  ratings,  and  the  rating  scales  often  are  not  detailed  enough 
to  accurately  reflect  the  behavior  being  evaluated.  However,  subjective  rating  scales  can 
be  useful  for  tasks  that  are  difficult  to  measure  such  as  decision  making  and  mood 
evaluations.  This  was  evident  in  a  study  by  Connolly,  Blackwell,  and  Lester  (1989) 
where  raters  assigned  a  subjective  score  to  the  pilot’s  decision  making  ability.  The 
researchers  found  this  a  useful  tool  for  evaluating  the  pilot’s  judgment — an  aspect  of 
aviation  job  performance  that  is  not  currently  amenable  to  more  objective  assessment.  In 
the  case  of  aircraft  control  skill,  most  evaluations  have  centered  on  the  use  of  objective 
measures  because  they  are  quantifiable  and  involve  identifiable  standards  against  which 
performance  can  be  measured  (Mixon  and  Moroney,  1981). 

The  United  States  Air  Force  focused  on  a  variety  of  objective  Automated 
Performance  Measurement  (APM)  tools  beginning  in  1968  in  an  attempt  to  increase  the 
objectivity  and  reliability  of  pilot  performance  measurement  (Fuller,  Waag,  and  Martin, 
1980).  The  APM  tools  became  of  particular  interest  in  training  environments  such  as  the 
Undergraduate  Pilot  Training  (UPT)  program  where  objective  evaluation  techniques  were 
needed  to  measure  the  student  pilot’s  performance  (Knoop  and  Welde,  1973).  The 


United  States  Army  also  sought  to  find  objective,  quantitative  methods  to  evaluate  pilot 
performance  in  rotary  wing  aircraft  (Billings,  Eggspuehler,  Gerke  and  Chase,  1968). 
Although  initial  efforts  to  precisely  quantify  piloting  skill  were  less  than  optimal,  better 
methods  for  collecting,  measuring,  and  analyzing  objective  flight  performance  data 
evolved  as  researchers  continued  to  explore  the  best  evaluation  techniques  to  measure 
pilot  performance  in  various  aircraft  (Mixon  and  Moroney,  1981). 

One  popular  method  is  to  calculate  the  standard  deviations  from  the  mean,  such  as 
used  by  Bisson  et  al.  (1993)  to  estimate  levels  of  pilot  precision  based  on  digital  flight 
data  collected  from  a  C-141 .  When  using  standard  deviations,  however,  information 
about  the  error  relative  to  a  given  set  of  criteria  is  not  provided.  Instead,  the  standard 
deviation  technique  measures  how  widely  values  are  dispersed  from  the  average  value 
(which  may  or  may  not  be  centered  on  the  correct  flight-control  parameter).  Root  mean 
square  errors  (RMSE),  on  the  other  hand,  convey  the  deviation  from  clearly  established 
criteria  and  provide  an  assessment  of  how  well  the  pilot  meets  various  ideal  standards, 
such  as  maintaining  specific  target  altitudes,  airspeeds,  etc.  The  smaller  the  RMSE  value, 
the  better  the  performance.  Various  studies  have  used  RMSE  to  measure  pilot 
performance.  An  early  study  conducted  by  Billings,  Gerke,  and  Wick  (1975)  used 
absolute  error  and  RMSE  variability  to  compare  flight  performance  in  simulated  and 
actual  flight.  In  another  study,  Caldwell  and  Jones  ( 1 990)  used  RMSE  to  effectively 
evaluate  helicopter  pilot  performance  with  atropine  sulfate  during  actual  and  simulated 
flight.  As  evidenced  by  the  results  of  this  latter  study,  the  RMSE  approach  was  able  to 
accurately  establish  differing  levels  of  flight-performance  degradations  attributable  to  2-4 
mg  of  atropine  sulfate  in  comparison  to  a  nominal  placebo  condition. 
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Similar  to  the  methodology  used  by  Caldwell  and  Jones  (1990),  this  report 
describes  a  scoring  routine  that  uses  RMSE  to  measure  the  impact  of  fatigue  (from  sleep 
deprivation)  on  flight  performance.  This  paper  will  discuss  the  methods  used  to 
objectively  assess  and  quantify  the  flight  performance  of  F-l  17A  pilots  collected  using 
the  Link  F-l  17A  Weapon  System  Training  (WST)  (L3  Communications,  1993)  device  at 
Holloman  AFB.  The  flight  methodology  and  examples  discussed  in  this  paper  originated 
from  a  fatigue  study  entitled,  “The  Effects  of  37  Hours  of  Continuous  Wakefulness  on 
the  Alertness  and  Flight  Performance  of  F-l  17A  Pilots,”  in  which  the  effects  of  fatigue 
on  flight  performance  were  objectively  assessed  (Caldwell  et  al,  2003).  A  subsequent, 
follow-on  study,  conducted  with  a  drug  intervention,  also  employed  these  methods  to 
determine  the  extent  to  which  a  fatigue  countermeasure  mitigated  the  effects  of  untreated 
sleep  loss  on  simulator  control  accuracy  (Caldwell  et  al,  2004). 

METHODS 

Flight  Simulator 

The  F-l  17A  WST  at  Holloman  Air  Force  Base,  NM,  was  the  simulator  used  to 
collect  all  of  the  flight  performance  data.  The  simulator  is  a  motionless  device  that  provides 
a  fully-working  replica  of  the  cockpit  found  in  the  actual  F-l  1 7  aircraft,  including  all 
primary  and  secondary  flight  controls,  engine  sounds,  and  cockpit  lighting  (L-3 
Communications,  1 993).  It  is  designed  to  accurately  replicate  all  aspects  of  the  Air  Force  F- 
1 17A  aircraft,  including  aircraft  systems  and  operation.  The  components  of  the  WST 
include  the  simulator  itself  as  well  as  an  instructor/operator  station,  a  computer  complex  that 
includes  an  Alpha  Server  8200  and  Input/Output  cabinets,  and  the  equipment  necessary  for 
the  generation  of  out-of-the-window  and  1R  visual  scenes.  The  actual  F-l  17A  aircraft  is  a 
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twin-turbofan  powered,  low-radar,  ground-attack  fighter  with  a  single-seat  cockpit. 

Training  in  the  WST  is  directly  transferable  to  the  actual  F-l  17  aircraft  in  terms  of 
instrument  flights,  takeoffs  and  landings,  instrument  navigation,  system  operations,  and  air- 
to-ground  attack  procedures. 

Objective  flight  performance  data  were  collected  using  the  Coherent  Automated 
Simulation  Test  Environment  (CoASTE)  tool — a  set  of  software  routines  that  normally 
provides  the  capability  to  evaluate  simulator  performance,  display/manipulate  various  data 
from  simulator  data  pools,  and/or  trace  and  correct  problems.  The  CoASTE  tool’s  trace 
utility  was  used  to  capture  various  parameters  of  flight  performance  data  at  a  rate  of  2  Hz 
throughout  each  flight  (Caldwell  et  al.,  2003).  One  complete  data  file  was  generated  for 
each  simulator  flight.  This  file  contained  all  the  data  collected  from  the  beginning  to  the  end 
of  the  given  simulation  session.  Each  record  in  the  file  contained  the  time  at  which  each 
data  sample  was  collected,  the  actual  data  points  themselves,  and  an  identification  field 
which  consisted  of  the  subject  number,  testing  day,  and  testing  session.  The  completed  data 
files  were  downloaded  onto  a  700  MB  Compact  Disk  Recordable  (CD-R)  at  the  conclusion 
of  data  collection  before  being  transferred  to  a  standard  desktop  Pentium-based  computer 
where  the  data  were  further  reduced  and  analyzed,  a  process  that  will  be  extensively 
described  in  this  paper. 

Flight  Assessment 

For  this  particular  study,  the  primary  objective  was  tracking  the  impact  of  fatigue  on 
basic  piloting  skills.  Therefore,  it  was  crucial  that  a  standardized  flight  profile  be  flown  for 
all  the  simulator  flights.  The  flight  profile,  as  shown  in  Table  1,  was  comprised  of  15 
standardized  instrument  flight  maneuvers  that  were  repeated  in  the  same  sequence  at  5-hour 
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intervals  throughout  the  final  portion  of  a  37-hour  period  of  continuous  wakefulness.  There 
were  5  test  flights  during  this  time,  each  of  which  was  1-hour  in  duration.  The  flights  were 
conducted  at  2300,  0400,  0900,  1400,  and  1900. 

The  reason  the  flight  profile  shown  below  was  chosen  was  because  it  tapped  the 
basic  “stick  and  rudder”  skills  required  of  all  pilots,  and  it  permitted  precise  and  quantifiable 
assessment  of  pilot  performance  across  all  participants  in  the  study.  Ifnon-standardized 
maneuvers  had  been  chosen,  it  would  have  been  difficult  to  assess  performance  because 
pilots  often  use  different,  but  equally  correct,  techniques  to  successfully  accomplish  the 
same  mission.  A  pilot’s  ability  to  fly  the  types  of  precision  instrument  maneuvers  used  here 
enables  a  general  assessment  of  how  they  would  be  able  to  react  to  more  complex,  real 
world  situations. 

_ Table  1 _ Flight  maneuvers _ 

Number  Detailed  maneuver  descriptions 

L  Right  360°  turn  at  an  altitude  of  1 1,000  ft  mean  sea  level  (MSL) 

2.  Straight  and  level  on  a  heading  of  345°  at  1 1 ,000  ft  MSL 

3.  Left  360°  turn  at  an  altitude  of  1 1 ,000  ft  MSL 

4.  Straight  climb  from  1 1,000  to  13,000  ft  MSL 

5.  Straight  and  level  on  a  heading  of  345°  at  13,000  ft  MSL 

6.  Descending  right  360°  turn  to  an  altitude  of  10,000  ft  MSL 

7.  Straight  and  level  on  a  heading  of  345° 

8.  Left-climbing  540°  turn  to  an  altitude  of  1 5,000  ft. 

9.  Straight  and  level  on  a  heading  of  165°  at  15,000  ft  MSL 

1 0.  Right  360°  turn  at  an  altitude  of  1 5,000  ft  MSL 

1 1 .  Straight  and  level  on  a  heading  of  1 65°  at  1 5,000  ft  MSL 

1 2.  Left  720°  turn  at  an  altitude  of  1 5,000  ft  MSL 

13.  Straight  descent  from  15,000  to  13,000  ft  MSL 

14.  Intercept  localizer  (not  scored) 

15.  Instrument  Landing  System  (ILS)  approach  to  Runway  16 
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Prior  to  the  sleep-deprivation  period  in  which  the  test  flights  were  flown,  each  pilot 
was  trained  to  asymptote  on  the  standardized  profile  via  the  completion  of  three  training 
flights.  Upon  arrival  for  training  and  for  participation  in  the  study,  the  participant  was  given 
a  pre-flight  mission  briefing  by  a  member  of  the  research  staff  who  served  as  the  console 
operator  during  all  of  the  initial  training  flights.  The  research  staff  member  informed  the 
participant  about  the  flight  maneuvers  that  would  be  flown  and  the  manner  in  which  each 
flight  would  be  conducted.  The  pilot  was  told  that  a  staff  member  would  explicitly 
sequence  them  through  each  of  15  flight  maneuvers  at  the  appropriate  point  in  each  of  the 
flight  profiles.  The  participants  were  instructed  that  they  would  be  required  to  establish 
flight  parameters  within  3°  of  the  target  heading,  3  knots  of  the  target  airspeed,  and  10  ft 
of  the  target  altitude  before  each  maneuver  began.  This  was  done  to  ensure  accurate  data 
measurement  and  also  to  increase  the  validity  of  using  the  performance  metric  of  RMSE 
to  measure  flight  performance.  RMSE  has  limitations  in  measuring  deviations  from  an 
assigned  flight  path  when  large  deviations  from  the  target  value  occur;  therefore,  it  was 
critical  that  the  specified  parameters  were  met  before  the  maneuver  commenced 
(Hubbard,  1987).  The  pilot  was  encouraged  to  precisely  fly  all  of  the  flight  parameters 
and  to  maintain  strict  standards  of  performance  throughout  each  of  the  flights. 

Once  the  training  flights  were  completed,  the  pilot  participant  was  released  and 
given  a  night  off  prior  to  the  sleep-deprivation  period  which  began  on  the  following  day. 
Upon  return  to  the  simulator  facility  (the  next  day),  the  pilot  completed  the  5  flights 
described  earlier  (at  2300,  0400,  0900,  1400  and  1900).  This  same  procedure  was  used 
for  all  10  pilots  who  were  tested  in  the  study. 
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DATA  COLLECTION 

The  flight  data  were  collected  with  the  CoASTE  tool  and  collected  as  a  continuous 
stream  from  beginning  to  the  end  of  each  flight.  The  CoASTE  tool  captured  several 
parameters  of  flight  performance  data  (see  Table  2)  at  a  rate  of  2  samples  recorded  every 
second  during  each  flight  maneuver. 

Table  2.  Measured  simulator  flight  parameters 
Parameter 

1 .  Indicated  altitude  (feet) 

2.  Indicated  airspeed  (knots) 

3.  Indicated  vertical  speed  (feet  per  min.) 

4.  Magnetic  heading  (degrees) 

5.  Pitch  angle  (degrees) 

6.  Roll  angle  (degrees) 

7.  Slip  ((ball  widths) 

8.  Localizer/course  deviation  (dots) 

9.  Glideslope/course  deviation  (dots) 

The  data  from  the  CoASTE  tool  were  saved  as  an  ASCII  file  and  the  data  were  then 
imported  into  Microsoft  Excel  for  subsequent  data  reduction.  The  next  step  was  to  organize 
the  data  by  splitting  it  into  the  15  maneuvers  based  on  the  mission  elapsed  time  that  was 
marked  by  the  console  operator  at  the  start  of  each  of  the  1 5  maneuvers  throughout  the 
flight.  Once  the  flight  data  had  been  split  into  the  1 5  maneuvers,  the  next  step  was  to  trim 
the  flight  data  based  on  the  parameters  for  each  maneuver  so  that  the  transition  into  and  out 
of  the  maneuver  was  removed.  For  example.  Maneuver  4  consisted  of  a  straight  climb  at  a 
rate  of  1000  feet  per  minute  (fpm).  Some  pilots  began  a  consistent  climb  quickly,  while 
others  took  longer  to  consistently  climb  at  1000  fpm.  To  eliminate  any  variability  in 
transitioning  from  level  flight  into  the  climb,  the  data  were  trimmed  so  that  Maneuver  4 
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started  when  the  pilots  displayed  a  consistent  climb  rate  for  several  consecutive  seconds  and 
ended  prior  to  the  transition  back  to  level  flight.  For  Maneuver  1 ,  a  right  360°  turn  at  a  30° 
angle  of  bank,  the  flight  data  were  trimmed  so  that  the  data  to  be  scored  for  the  maneuver 
was  limited  to  what  was  recorded  once  the  pilots  were  consistently  maintaining  a  30°  angle 
of  bank.  The  data  points  leading  up  to  the  30°  mark  were  eliminated  to  reduce  variability 
amongst  pilots  and  to  ensure  that  only  the  heart  of  the  maneuver,  and  not  the  transition  into 
or  out  of  the  maneuver,  was  scored.  To  eliminate  the  ending  transition  (when  the  pilot  was 
rolling  out  of  the  turn)  the  roll-out  data  were  excluded  as  well.  Essentially  this  same  type  of 
procedure  was  used  on  all  of  the  scored  flight  maneuvers. 

For  each  maneuver,  the  3  most  relevant  parameters  were  scored.  This  issue  is 
further  discussed  below  using  the  left  720°  turn  (Maneuver  number  12)  as  an  example. 

Figure  1  provides  a  graphical  representation  of  the  raw,  trimmed  flight  data  collected 
on  the  10  pilots  for  the  left  720°  turn  across  the  3  parameters  that  were  scored  for  that 
maneuver:  altitude,  airspeed,  and  roll.  On  each  graph,  there  is  a  single  line  for  each 
individual  pilot.  If  the  pilots  had  flown  this  maneuver  perfectly,  there  would  be  one  solid 
horizontal  line  extending  (from  left  to  right)  from  the  target  values  of  1 5,000  ft,  300  knots, 
and  30°  angle  of  bank  since  every  collected  sample  would  have  corresponded  perfectly  to 
the  ideal  values;  however,  varying  degrees  of  deviations  occurred  amongst  the  pilots.  The 
particularly  large  amplitude  increases  from  the  target  values,  present  during  session  3  and  4, 
illustrate  how  flight  performance  deteriorated  over  time. 
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Altitude  deviation  from  1 5,000  ft 


2300  -  Test  Session  1  0400  -  Test  Session  2  0900  -  Test  Session  3  1400  -  Test  Session  4  1  900  -  Test  Session  5 

imm - 1 1S4M - 1  nmm - i  imn - ,  nitt 


Airspeed  deviation  from  300  knots 

2300  -  Test  Session  1  0400  -  Test  Session  2  0900  -  Test  Session  3  1400  -  Test  Session  4  1900  -  Test  Session  5 
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Roll  deviation  from  30° 

2300 -Test  Session  1  0400  -  Test  Session  2  0900  -  Test  Session  3  1400 -Test  Session  4  1900  -  Test  Session  5 


Figure  1 .  Ten  pilots’  raw  data  on  left  720°  turn  maneuver  across  altitude,  airspeed,  and  roll. 


DATA  REDUCTION 

Root  Mean  Square  Error  (RMSE) 

The  next  step  was  to  calculate  the  RMSE  for  each  scored  flight  parameter  for  each 
maneuver  (the  parameters  are  shown  in  Table  3).  Different  parameters  were  evaluated 
for  each  maneuver  because  of  the  relevance  of  those  parameters  to  the  proper  conduct  of 
specific  flight  tasks.  For  example,  the  altitude  on  a  climb  was  constantly  changing 
because  this  is  the  nature  of  a  climb;  therefore,  altitude  deviation  would  have  been  a 
meaningless  measure  in  this  case  whereas  a  better  parameter  to  score  pilot  control 
technique  was  the  vertical  velocity  because  it  revealed  a  pilot’s  ability  to  maintain  a 
consistent  climb  rate. 
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Table  3.  Maneuver’s  flight  parameters 


Detailed  maneuver  descriptions  Evaluated  parameters 


1.  Right  360°  turn 

2.  Straight  and  level  1 

3.  Left  360°  turn 

4.  Straight  climb 

5.  Straight  and  level  2 

6.  Descending  right  360°  turn 

7.  Straight  and  level  3 

8.  Left-climbing  540°  turn 

9.  Straight  and  level  4 

10.  Right  360°  turn 

1 1 .  Straight  and  level  5 

12.  Left  720°  turn 

13.  Straight  descent 

14.  Intercept  localizer 

15.  Instrument  Landing  System  (ILS) 


altitude,  airspeed,  roll 
altitude,  airspeed,  heading,  roll 
altitude,  airspeed,  roll 
airspeed,  vertical  velocity,  heading 
altitude,  airspeed,  heading,  roll 
airspeed,  vertical  velocity,  roll 
altitude,  airspeed,  heading,  roll 
airspeed,  vertical  velocity,  roll 
altitude,  airspeed,  heading,  roll 
altitude,  airspeed,  roll 
altitude,  airspeed,  heading,  roll 
altitude,  airspeed,  roll 
airspeed,  vertical  velocity,  heading 
not  scored 


RMSE  was  calculated  using  the  following  formula: 

~n - 


RMSE  = 


i  =  1  (Target  Value  -  Observed  Value) 
n 


Table  4  shows  how  the  RMSE  formula  was  applied  to  a  pilot’s  data  on  the  left  720° 
turn  maneuver.  The  observed  altitude  values  were  subtracted  from  the  target  altitude  of 
1 5,000  ft.  The  score  was  squared  and  summed  across  the  entire  maneuver  to  yield  a  value 
of  3, 1 20,806.3 1 .  Next,  the  result  was  divided  by  the  total  number  of  samples  (829) 
collected  from  a  single  pilot  during  the  maneuver,  and  the  square  root  of  the  result  was 
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taken  to  yield  the  RMSE  of  61.36  ft.  Therefore,  on  this  particular  maneuver,  the  pilot 
averaged  an  error  of  61.36  ft  as  he  attempted  to  maintain  an  altitude  of  15,000  ft. 


Table  4.  Left  720°  turn  -  Altitude  RMSE 


Time  (s) 

Altitude  (ft) 

Tarqet  Altitude  (ft) 

Dev  A2 

1948.0 

15002.9 

15000 

8.41 

1948.5 

15000.9 

15000 

0.81 

1949.0 

14999.8 

15000 

0.04 

1 

1 

i 

i 

2361.0 

14975.7 

15000 

590.49 

2361.5 

14973.1 

15000 

723.61 

2362.0 

14970.3 

15000 

882.09 

N 

Sum  DevA2 

829 

3120806.31 

RMSE 

61.36 

Figure  2  represents  the  RMSE  values  that  were  calculated  using  the  raw,  trimmed 
flight  data  shown  in  Figure  1.  Thus  each  data  line  in  Figure  2  has  now  been  reduced  to  a 
single  number.  The  RMSE  values  for  the  left  720°  turn  were  calculated  independently 
for  altitude,  airspeed,  and  roll,  across  the  five  testing  sessions  (yielding  one  data  point  per 
subject  per  session).  For  all  the  subsequent  graphs  (one  each  for  the  3  scored  parameters) 
the  10  thin  lines  represent  the  10  pilots,  while  the  thick  line  represents  the  composite 
average  performance  across  the  10  pilots.  The  graphs  provide  a  useful  visual  aid  in 
viewing  how  errors  increased  over  time  as  the  pilots  attempted  to  maintain  target  values 
of  altitude,  airspeed,  and  roll  on  the  left  720°  turn  despite  growing  amounts  of  sleep 
deprivation.  Note  that  this  same  procedure  was  used  to  characterize  performance  on  the 
other  maneuvers  as  well. 
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Altitude  RMSE 


Airspeed  RMSE 


Roll  RMSE 


Figure  2.  Ten  pilots’  RMSE  scores  on  left  720°  turn  maneuver  across  altitude,  airspeed, 
and  roll. 

Percent  Change 

Since  the  flight  parameters  were  all  represented  in  various  units  of  measurement 
such  as  feet,  knots,  degrees,  etc.,  the  conversion  to  percent  change  scores  enabled 
multiple  flight  parameters  to  be  compared  on  the  same  scale.  Consequently,  this  enabled 
a  composite  score  to  be  calculated  for  each  maneuver  and  this  made  it  possible  to 
compare  performance  across  different  times  and  across  different  maneuvers. 

The  conversion  from  RMSE  into  percent-change-from-baseline  scores  was 
accomplished  using  the  following  formula: 

Percent  Change  =  ((Baseline  RMSE  -  Deprivation  RMSE)/Baseline  RMSE)  x  100.00 
For  example.  Table  5  shows  how  the  altitude  percent  change  was  calculated  for  one  pilot 
on  the  left  720°  turn.  The  baseline  scores  were  obtained  from  the  third  training  flight 
conducted  the  day  prior  to  the  testing  (sleep  deprivation)  period.  For  the  2300  session, 
the  testing  altitude  RMSE  of  61.36  ft  was  subtracted  from  the  baseline  altitude  RMSE  of 
69.30  ft.  The  result  was  then  divided  by  the  baseline  score  and  multiplied  by  100  to  yield 
a  percent  change  of  -1 1 .46%.  The  same  series  of  calculations  were  performed  for  the 
subsequent  test  sessions  at  0400,  0900,  1400,  and  1700,  and  this  procedure  was  followed 
for  all  of  the  evaluated  parameters  for  each  maneuver. 
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Table  5.  Left  720°  turn  -  Altitude  percent  change 


Time 

Baseline 

Testing 

Percent 

Altitude  RMSE 

Altitude  RMSE 

Change 

2300 

69.30 

61.36 

-11.46% 

400 

69.30 

80.77 

16.56% 

900 

69.30 

98.66 

42.37% 

1400 

69.30 

83.58 

20.61% 

1900 

69.30 

59.73 

-13.81% 

The  conversion  of  the  RMSE  into  percent  change  scores  revealed  the  decrease  (or 
increase)  in  flight  performance  over  the  sleep  deprivation  period.  Figure  3  shows  the 
percent  change  scores  that  were  calculated  based  on  the  altitude,  airspeed,  and  roll  RMSE 
data  shown  in  Figure  2  (again,  using  the  Left  720°  turn  as  an  example). 


Altitude  Percent  Change 


Airspeed  Percent  Change 


Roll  Percent  Change 


Time  Time  Time 


Figure  3.  Ten  pilots’  percent  change  scores  on  left  720°  turn  maneuver  across  altitude, 

airspeed,  and  roll. 

Once  each  scored  parameter  was  converted  from  RMSE  to  percent  change,  a 
composite  flight  performance  score  was  calculated  for  each  of  the  flight  maneuvers.  For 
the  left  720°  turn  the  altitude,  airspeed,  and  roll  percent  change  scores  were  averaged 
together  at  each  time  point.  This  was  accomplished  for  each  subject.  As  shown  in  Figure 
4,  the  composite  percent  change  graph  reveals  the  similar  trend  in  performance  seen  in 
the  individualized  altitude,  airspeed,  and  roll  percent  change  graphs  (Figure  3),  except 
that  greater  errors  are  now  represented  by  a  downward  deflection  (all  of  the  percent 
change  data  were  inverted  to  aid  in  data  interpretation  (since  this  graphs  poor 
performance  as  “down”  and  good  performance  as  “up”)). 
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Left  720  Turn 

Composite  Percent  Change 


Figure  4.  Ten  pilots’  composite  percent  change  scores  on  the  left  720°  turn  maneuver. 

In  addition  to  the  left  720°  turn,  the  composite  percent  change  scores  were 
calculated  across  the  additional  flight  maneuvers  seen  in  Figure  5:  right  360°  turns, 
straight  and  levels,  right  descending  turn,  left  climbing  turn,  left  360°  turn,  climb,  and 
descent.  The  same  series  of  steps  described  above,  with  the  left  720°  turn,  were  applied 
to  these  maneuvers.  Note  that  multiple  iterations  of  the  straight  and  levels  and  the  right 


360°  turns  have  been  averaged  together  to  produce  only  one  graph  per  maneuver  type. 


Right  360  Turns 


Straight  and  Levels 


Right  Descending  Turn 
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Figure  5.  Ten  pilots’  composite  percent  change  scores  on  the  right  360°  turns,  straight 
and  levels,  right  descending  turn,  left  climbing  turn,  left  360°  turn,  climb,  and 
descent. 

To  further  reduce  the  amount  of  flight  data,  composite  scores  for  each  of  the  10 
pilots  were  calculated  by  averaging  their  individual  percent  change  scores  across  the  8 
maneuver  types  at  each  time  point.  Figure  6  depicts  the  10  pilots’  flight  performance 
scores  averaged  across  all  maneuvers  and  reveals  the  individual  variations  in  fatigue 
vulnerability  across  the  pool  of  10  pilots.  Note  that,  once  again,  the  thin  lines  represent 
individuals  while  the  thick  line  represents  the  composite  average  performance  across  the 
10  pilots. 


Flight  Performance  across  all  Maneuvers 


Figure  6.  Ten  pilots’  composite  percent  change  scores  across  all  maneuvers. 
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DATA  ANALYSIS 


Once  the  flight  data  were  transformed  into  percent  change  scores  for  each  pilot  at 
each  time  point  within  each  maneuver,  the  data  were  analyzed  with  BMDP4V,  Repeated 
Measures  Analysis  of  Variance  (ANOVA).  Also,  BMDP4V  was  used  to  conduct 
regression  evaluations  on  all  significant  effects  for  the  presence  of  linear,  quadratic,  and 
cubic  trends.  Overall,  there  were  five  testing  sessions  and  eight  different  maneuver  types 
(combined  right  360°  turns,  combined  straight  and  levels,  right  descending  turn,  left 
climbing  turn,  left  360°  turn,  climb,  left  720°  turn,  and  descent).  One-way  ANOVA ’s 
were  performed  on  the  eight  maneuver  types.  Figure  7  graphically  depicts  the  composite 
flight  performance  across  the  10  pilots  (averaged  together)  for  each  of  the  eight  maneuver 
types,  with  linear,  quadratic,  and  cubic  trends  annotated  (bottom  right  comer  of  each 
graph). 


Right  360  Turns 


Straight  and  Levels 


2300  400  900  1400  1900 

Time 


Right  Descending  Turn 


Left  Climbing  Turn 


Left  360  Turns 


Climb 


Left  720  Turn 


Descent 


Figure  7.  Ten  pilots’  composite  percent  change  scores  on  the  right  360°  turns,  straight 
and  levels,  right  descending  turn,  left  climbing  turn,  left  360°  turn,  climb,  left 
720°  turn,  and  descent  with  linear,  quadratic,  and  cubic  trends  annotated. 


Next,  an  overall  analysis  was  conducted  using  a  two-way  ANOVA  in  which  the 
factors  of  interest  were  testing  time  and  flight  maneuver.  As  shown  in  Figure  8,  there 
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was  a  time  main  effect  (F(2.44,21.93)=10.72,  p=.0003).  Subsequent  analyses  indicated 
this  to  be  a  function  of  significant  linear,  quadratic,  and  cubic  (p<.05)  trends.  Figure  8 
reflects  the  general  pattern  seen  with  the  various  flight  maneuvers,  with  performance 
accuracy  seriously  deteriorating  after  27-32  hrs  of  wakefulness. 

Composite  Flight  Performance  across  all  Maneuvers 


Figure  8.  Ten  pilots’  composite  flight  performance  across  all  maneuvers  with  linear, 
quadratic,  and  cubic  trends  annotated. 

CONCLUSION 

As  shown  in  this  paper,  the  evaluation  of  objectively-measured  flight  performance 
requires  a  precisely  structured  methodology  in  order  to  obtain  meaningful  and  useful 
results.  Past  researchers  have  developed  various  techniques  to  try  to  evaluate  pilot 
performance,  including  using  subjective  ratings  (Rehmann,  1982),  standard  deviations 
(Bisson  et  al.,  1993),  absolute  error,  and  RMSE  variability  (Billings  et  al.,  1975).  In  the 
case  of  the  study  outlined  here,  the  RMSE  technique  was  used  because  it  permitted  a 
clear  determination  of  the  extent  of  pilot  performance  degradations  associated  with 
lengthy  periods  of  continuous  wakefulness  (37  hours).  Such  data  are  useful  for 
substantiating  the  fact  that  this  particular  stressor  (sleep  deprivation)  does  in  fact 
compromise  the  operational  readiness  of  even  well-trained  military  professionals.  In  the 
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future,  these  data  may  form  the  basis  for  comparing  the  effectiveness  of  a  recommended 
sleep-deprivation  countermeasure,  and/or  they  may  be  used  to  help  design  work 
schedules  in  which  critical  tasks  are  avoided  during  particularly  dangerous  times. 

Regardless  of  specific  future  applications,  the  procedures  used  to  quantify  these 
results  were  shown  to  be  of  significant  value,  and  it  is  hoped  that  the  descriptive 
procedural  information  provided  here  will  enable  others  to  take  advantage  of  a  useful 
analytical  approach.  The  methodology  outlined  in  this  paper  can  be  applied  not  only  to 
the  study  of  pilot  fatigue,  but  to  the  assessment  of  other  stressors  or  interventions  that 
may  either  degrade  or  enhance  aviator  performance,  such  as  chemical  defense  antidotes 
(Caldwell  and  Jones,  1990),  alcohol  (Billings  et  al,  1991),  modafinil(Caldwell  et  al, 
2003),  or  others.  Clearly  the  computerized  RMSE  approach  offers  the  advantages  of 
objectivity,  repeatability,  and  interpretability,  while  avoiding  the  subjectivity, 
unreliability,  and  inherent  biases  that  can  adversely  impact  human  ratings. 
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APPENDIX  B 

Holloman  Simulator  Fatigue  Study  Flight  Script 


Subject  ID _ Day _ Session _ Time _ Date _ Tech  Initials _ 

_ Turbulence  0;  MEA  0;  Night  Ilium.;  ALT  1 1,000;  ASP  300;  HDG  345;  Disable  Autopilot/Auto-throttle 

_ SD  off,  I  RADS  off  _ Printer  paper  full 

_ Simulator  room  lights  off  _ Simulator  room  door  closed 

_ COASTE  initialized,  PI  LOT  ALT  SET  30.02  _ Radio  check 

_ Simulator  canopy  closed  _ EEG  Eyes-Open/Eyes-Closed 

_ Take  simulator  off  freeze/PRINT  PAGE  _ Initial  start  time  of  simulator _ 

Stabilize  flight  path,  straight  and  level,  at  1 1 ,000  feet,  heading  345  degrees,  airspeed  300  knots. 

Task  1.  On  my  mark,  your  first  maneuver  will  be  a  30-degree-angle-of-bank,  right  360-degree  turn,  from  a 
heading  of  345  degrees  to  345  degrees.  Maintain  1 1,000  feet  and  300  knots... Ready... Mark.  (After  mark 
repeat:  turning  right  to  a  heading  of 345  degrees,  30-degree  angle  of  bank,  at  11,000 feet,  300  knots). 

Task  2.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  2  minutes  of  straight-and-level  flight. 
Maintain  a  heading  of  345  degrees,  altitude  of  1 1,000  feet,  and  airspeed  of  300  knots... Ready... Mark.  (After 
mark  repeat:  maintain  heading  of 345  degrees,  1 1,000 feet,  300  knots). 

Task  3.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  30-degree-angle-of-bank,  left  360- 
degree  turn,  from  a  heading  of  345  degrees  to  345  degrees.  Maintain  1 1,000  feet  and  300  knots... 

Ready. ..Mark.  (After  mark  repeat:  turning  left  to  a  heading  of 345  degrees,  30-degree  angle  of  bank,  at  1 1,000 
feet,  300  knots). 

Task  4.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  straight  climb  on  a  heading  of  345 
degrees  from  1 1,000  feet  to  13,000  feet  at  1000  feet  per  minute.  Maintain  300  knots... Ready... Mark.  (After 
mark  repeat:  climbing  from  1 1,000  to  13,000 feet,  1000 feet  per  minute,  heading  345  degrees,  airspeed  300 
knots). 

Task  5.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  2-minute  straight  and  level. 
Maintain  a  heading  of  345  degrees,  altitude  of  13,000  feet,  and  airspeed  of  300  knots... Ready... Mark.  (After 
mark  repeat:  straight  and  level,  on  a  heading  of 345  degrees,  at  13,000 feet,  300  knots). 

Task  6.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  30-degree-angle-of-bank,  right 
descending  turn,  360  degrees,  from  a  heading  345  to  heading  345.  Descend  from  13,000  feet  to  10,000  feet  at 
1 000  feet  per  minute.  Maintain  300  knots. . .  Ready. . . Mark.  (After  mark  repeat:  30-degree-angle-of-bank,  right 
descending  360-degree  turn  to  a  heading  of 345  degrees.  Descending  to  10,000 feet  at  1000 feet  per  minute, 
maintain  300  knots). 
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Task  7.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  2-minute  straight  and  level. 
Maintain  a  heading  of  345  degrees,  altitude  of  10,000  feet,  and  airspeed  of  300  knots... Ready... Mark.  (After 
mark  repeat:  straight  and  level,  on  a  heading  of 345  degrees,  at  10,000  feet,  300  knots). 

(Check  EEG  traced 

Task  8.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  30-degree-angle-of-bank,  left 
climbing  turn,  540  degrees,  from  a  heading  345  to  heading  165  degrees.  Climb  from  10,000  feet  to  15,000  feet 
at  1000  feet  per  minute.  Maintain  300  knots... Ready... Mark.  (After  mark  repeat:  30-degree-angle-of-bank,  left 
climbing  540-degree  turn  to  a  heading  of  165  degrees.  Climbing  to  15,000 feet  at  1000 feet  per  minute, 
maintain  300  knots). 

Task  9.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  2-minute  straight  and  level. 
Maintain  a  heading  of  165  degrees,  altitude  of  15,000  feet,  and  airspeed  of  300  knots... Ready... Mark.  (After 
mark  repeat:  straight  and  level,  on  a  heading  of  165  degrees,  at  1 5,000 feet,  300  knots). 

Task  10.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  30-degree-angle-of-bank,  right 
360-degree  turn,  from  a  heading  of  165  to  165  degrees.  Maintain  15,000  feet  and  300  knots... Ready... Mark. 
(After  mark  repeat:  turning  right  to  a  heading  of  165  degrees,  30-degree  angle  of  bank,  at  15,000 feet,  300 
knots). 

Task  11.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  2-minute  straight  and  level. 
Maintain  a  heading  of  165  degrees,  altitude  of  15,000  feet,  and  airspeed  of  300  knots... Ready... Mark.  (After 
mark  repeat:  straight  and  level,  on  a  heading  of  165  degrees,  at  15,000 feet,  300  knots). 

Task  12.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  30-degree-angle-of-bank,  left  720- 
degree  turn,  from  a  heading  of  165  to  165  degrees.  Maintain  15,000  feet  and  300  knots... Ready... Mark.  (After 
mark  repeat:  720-degree  left  turn  to  a  heading  of  165  degrees,  30-degree  angle  of  bank,  at  15,000 feet,  300 
knots). 

Task  13.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  straight  descent  on  a  heading  of 
165  degrees  from  15,000  feet  to  13,000  feet  at  1000  feet  per  minute.  Maintain  300  knots... Ready... Mark. 

(After  mark  repeat:  descending  from  15,000  to  13,000 feet,  1000 feet  per  minute,  maintain  heading  of  165 
degrees,  airspeed  300  knots). 

Task  14.  In  about  20  seconds  and  on  my  mark,  your  next  maneuver  will  be  a  straight  and  level,  maintaining 
13,000  feet,  slowing  to  250  knots. ...Ready. ..Mark.  (After  mark  repeat:  straight  and  level,  maintaining  13,000 
feet,  stowing  to  250  knots). 

Approaching  250  knots. 

You  are  now  cleared  direct  Higgy  (destination  14)  for  the  High  ILS,  Yankee,  Runway  16,  Low  Approach, 
maintain  13,000  feet. 

Once  turned  towards  Higgy  (within  5-10  miles  RNG). 

You  are  now  cleared  for  the  High  ILS,  Runway  16,  report  departing  Higgy  and  13,000  feet,  maintaining  250 
knots. 

Once  pilot  reports  departing  Higgy  (turning  to  heading  213.  descending  out  of  13.000  feet). 

Say:  Cleared,  High  ILS,  Runway  16. 


23 


Task  15.  Once  the  pilot  intercepts  the  localizer.  MARK. 

Report  5  miles  with  the  gear  and  slow  to  final  approach  airspeed  once  you  are  configured  with  the  gear  down. 
Once  the  pilot  gives  the  gear  check. 

You  are  cleared  low  approach.  Runway  16,  report  going  missed  approach  at  Decision  Height.  (DH  is  4283 
feet) 

Once  the  pilot  reports  going  missed  approach. 

MARK  (  This  is  the  end  of  the  flight;  give  it  a  few  seconds  then  put  the  sim  on  freeze  and  end  this  segment  of 
the  test  by  telling  the  pilot  to  go  ahead  and  get  out  or  standby  while  you  get  the  leads  disconnected. 


24 


APPENDIX  C 
Holloman  Fatigue  Study 
Participant  Pre-flight  Briefing 

1 .  There  will  be  a  total  of  8  flights  in  the  study. 

2.  Three  training  flights  on  day  1 ;  Five  test  flights  on  day  2. 

3.  You  may  be  coached  by  an  IP  on  the  training  flights,  but  not  the  test  flights. 

4.  The  flight  profile  consists  of  15  instrument  maneuvers  including  an  ILS  to  a 
missed  approach. 

5.  All  maneuvers  will  be  flown  at  night  with  no  illumination. 

6.  Start  and  Stop  points  for  each  maneuver  will  be  specified  by  the  console  operator 
throughout  the  study,  so  please  don 7  start  any  maneuver  until  instructed  to  do  so. 

7.  We  will  inform  you  about  your  next  maneuver  about  20  seconds  prior  to  the  end 
of  the  maneuver  you  are  presently  flying.  However,  you  should  continue  with  the 
current  maneuver  until  it  is  complete.  You  are  responsible  for  rolling  out  on  the 
correct  headings  and  altitudes. 

8.  Flight  tolerances  are  much  closer  than  60-2  standards.  Performance  will  be 
measured  to  exacting  standards  by  computer.  For  example — heading  within  1 
degree,  altitude  within  1  foot,  etc. 

9.  Autopilot  and  autothrottle  are  not  to  be  used;  however,  you  can  use  the  DEP  and 
INS  for  steer  modes,  instrument  reference,  and  destination  points. 

10.  The  objective  is  to  be  as  perfect  as  possible  at  all  times. 

1 1 .  Because  of  the  EEG  and  Eye  Tracking,  you  can’t  chew  gum  or  candy  during  the 
flights,  and  you  should  try  to  keep  your  jaw  muscles  and  face  muscles  relaxed. 

12.  Also,  because  of  the  EEG  and  Eye  Tracking,  conversation  must  be  kept  to  a 
minimum  at  all  times.  It  is  not  necessary  to  repeat  back  instructions  for  the 
maneuvers.  However,  during  the  ILS,  we  will  ask  you  to  call  out  some  report 
points. 

13.  During  training  flights,  you  can  talk  to  ask  questions,  get  clarifications,  etc. 
because  we  want  you  to  get  up-trained  as  well  as  possible  on  the  flight  profile. 

14.  During  the  testing  flights,  we  ask  that  you  remain  as  quiet  as  possible,  but  don’t 
hesitate  to  ask  us  to  repeat  instructions  if  you  forget  something. 

1 5.  Make  sure  you  have  the  approach  book  with  you  into  the  simulator.  For  the  ILS, 
switch  to  destination  14,  go  to  Higgy. 

16.  Once  in  the  sim,  we  will  hook  up  the  equipment,  turn  off  the  lights,  close  the 
canopy,  do  a  radio  check,  then  do  an  eyes-open/eyes-closed  EEG,  take  the  sim  off 
freeze,  and  we’ll  go. 

Show  the  participant  the  flight  profile  and  give  him  examples  of  the  instructions. 

If  the  participant  asks  what  call  sign  to  use  during  the  profde,  tell  him  SIM 
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